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(g) Audio user Interim with stereo and filterad sound effects for visually impaired users. 



@ Disclosed is a computer audio interface hav- 
Ing stereo and filtered sound effects to enable 
blind users to operate a graphical user Inter- 
fece. Stereo balance and incremental filtering 
are used along separate axes to guide a blind or 
visually impaired user within an area of a 
graphical user interface, particularly the dient 
area of a window. As the pointer approaches the 
left boundary of the client area, the sounds 
representing the dient area come more and 
more exduslvely from the left audio channel. 
Likew^, when approaching the right bound- 
ary, the sound shifts to the right channel. Ad- 
ditionally, as the pointer is moved toward the 
top of the window dient area, the pitch of the 
sound increases in stepwise fashion. 
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The present invention relates generally to conv 
puter system user interfaces and nrK>re particularly to 
an audio interface having stereo and filtered sound 
effects for enabling blind or visually impaired users to 
operate a computer system with a giaphical user in- 
terface. 

In recent years, there has been a move among 
computer application software developers toward 
graphical user interfaces. In graphical user interne- 
es, objects are presented for users to manipulate In 
ways that are similar to the way that they are manipu- 
lated in the real work place. Objects, such as f Qe cab- 
inets, folders, documents, and printers, are displayed 
on the screen as icons. Users manipulate these ot>- 
jects with a mouse to perform desired operations. For 
example, to file a document in a folder that is located 
in a file cabinet in the real work place, the user opens 
the f Qe cabinet, locates and opens the correct folder, 
and puts the document Inside. In the electronic work 
place of the graphical user interface, the user per- 
forms a similar process. The user opens the file cab- 
inet icon, locates the correct folder icon, and drops 
the document icon in the folder. Because this is an 
electronic environment, users do not have to open the 
folder to put the document in it However, users have 
been able to use their knowledge of a real work place 
to perform this operation. 

Normally sighted persons find graphical user in- 
terfaces intuitive and easy to work with. However, ex- 
cept for an occasional "beep" or "bong", graphical 
user interfaces are virtually silent and the vast major- 
ity of the information they provide to the user is vis- 
ual. Thus, graphical user interfaces are essentially 
not usable by blind or severely visually impaired peo- 
ple. 

Blind and visually impaired computer users now 
benefit from many forms of adaptive technology, In- 
cluding speech synthesis, large-print processing, 
braille desktop publishing, and voice recognition. 
However, presently, almost none of the foregoing 
tools is adapted for use with graphical user interfaces. 
It has been suggested that programmers could write 
software with built-in voice labels for icons. Lazzaro, 
Windows of Vulnerability, Byte Magazine, June 1991, 
page 416. Various synthetic or recorded speech sol- 
utions for making computer display screen contents 
available to blind persons have been suggested, for 
example in Golding, et al., IBM Technical Disdosure 
Bulletin, Vol. 26, No. 10B, pages 5633-5636 (March 
1984), and Barnett, et. al., IBM Technical Disclosure 
Bulletin, Vol. 26, No. 10A. pages 4950-4951 (March 
1984). Additionally, there have been suggested sys- 
tems that include a mouse with a braille transducer so 
that a blind user may read text and obtain certain tac- 
tile position feedback from the mouse. Comerford, 
IBM Technical Disclosure Bulletin, Vol. 28, No. 3, 
page 1343 (August 1985). Affinito, et al., IBM Tech- 
nical Disclosure Bulletin, Vol. 31. No. 12, page 386 



(May 1989). However, while announcing various text 
items, eit her audibly or by means of a braille transduc- 
er in the mouse, may provide some Information to 
blind user, it does not enable the user to navigate 
5 about and locate objects on the conriputer display 
screen. 

There has been suggested an audible cursor pos- 
itioning and pbcel (picture element) status identifica- 
tion mechanism to help a user of an interactive com- 

10 puter graphics system locate data by using aural feed- 
back to enhance visual feedback. As the cursor is 
stepped across the screen, an audible dick is gener- 
ated that varies in tone corresponding in tone to the 
current status of each pbcel encountered. With this 

15 combination in audible and visual cursor feedback. It 
becomes a simple task to identify the desired line by 
noting the change in tone as the cursor nwves. For 
color display applications, each color is represented 
by a distinct tone so any single pbcel may be dlstln- 

20 guished from the surrounding pbcels of a different col- 
or. It has been suggested that t his system is especial- 
ly helpful for visually impaired or learning disabled 
users. Drumm. et at.. IBM Technical Disdosure Bul- 
letin, Vol. 27. No. 48, page 2528 (September 1984). 

25 However, the foregoing disdosure does not suggest 
a means of enabling a blind user to navigate about or 
locate objects on the computer display screen. 

In the present invention, a stereo balance effect 
is used to convey Information about the position of 

30 the pointer in the left/right or Xdlrectfon relative tothe 
limits of the dient area of the current window. The 
system of the present inventfon indudes laterally 
spaced apart audio transducers, which may be 
speakers or stereo headphones. As the pointer ap- 

35 proaches the left boundary of the dient area, the 
sounds representing the dient area come more and 
more exdusively from the left audio channel, tike- 
wise, approaching the right boundary causes the 
sound to shift to the right channel. Centering the poln- 

40 ter within the window causes equal sound outputfrom 
both stereo channels. This audio effect is dramatic 
and effective. It also allows the user to sense quickly 
the size of the window. If the user hears a large bal- 
ance shift for relathrely little mouse nwvement, the 

45 user can sense that the window is narrow. Addition- 
ally, in the present invention, a different effect is im- 
plemented to communicate relative position in the 
top/bottom or Y axis of the window dient area. In this 
aspect of the invention, the frequency of the sounds 

60 representing the dient is a function of the top/bottom 
or Y positton of the pointer within the window dient 
area. Preferably, the frequency is changed in a fbced 
number of discrete steps, which allow the user to 
count them and better ascertain top/bottom or Y pos- 

55 ition. In the preferred embodiment, the frequency is 
increased as the pointer moves from the bottom to 
the top of the client area, which fdlows the intuitive 
metaphor of high pitched sounds corresponding to a 
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high position. 

A particular problem that b\'m6 or visually im- 
pairad users have in operating graphical user inter- 
faces Is navigating in windows. Windows include a cli- 
ent area that is populated with text and or icons. 5 
Sighted users can find objects within windows at a 
glance and move the pointer to them almost without 
thinking. However, a blind or visually impaired user, 
even if provided with text to speech or other audio 
identification of the objects can find such objects only io 
through trial and error or random searching. More- 
over, it is. very difficult for a blind or visually impaired 
user, after having found and identified all of the ob- 
jects in the window, to navigate back to a desired ob- 
ject 15 

Figure 1 is a pictorial view of a window with rela- 
tive amplitude and frequency scales added to aid in 
understanding the invention. 

Figure 2 Is a block diagram showing a preferred 
system of the present Invention. 20 

Figure 3 is a block diagram showing a preferred 
implementation of the sound generator of the present 
inventton. 

Rgure 4 Is a flowchart of a preferred software im- 
plementatton of the present invention. 25 
Referring now to the drawings, and first to Figure 

I , a window is designated generally by the numeral 

I I . Window 11 is displayed on a computer system dis- 
play screen, as is well known to those skilled in the 

art Window 11 includes a window border 13, a title 30 
bar 15, an actk>n bar 17, and a dient area 19. Title bar 
15 includes, in addition to the title of the window, a 
system menu icon 21, and window-sizing icons 23 
and 25. System menu Icon 21 allows a user to display 
a pull-down menu containing the actions that the user 35 
can perform on the window. Window-sizing icon 23 
provides a ^st way to use a mouse or other pointing 
device to minimize the window, by reducing it to an 
icon. Conversely, window-sizing loon 25 provides a 
fast way for the user to maximize t he window to fill the 4o 
entire screen. 

Action bar 17 contains a list of the actions of an 
application. The user can cause the system to display 
a pulkJown menu under each item in action t>ar 17. 

Client area 19 comprises the remainder of win- 45 
dow 11 . Client area 1 9 is the focus of the users atten- 
tion and it is where the user is presented with the ob- 
ject or objects upon which the user wishes to work. As 
those skilled in the art and those familiar windows will 
recognize, the window client area is normally popu- so 
lated with text and/or icons. However, for purposes of 
clarity and illustration, client area 19 is shown to be 
empty. 

A pointer 27 Is shown within dient area 1 9. Poin- 
ter 27 is moveable about the screen by means of a 65 
mouse (not shown) or other pointing device. The user 
can move pointer 27 to various objects to select, 
open, or directly manipulate them. People with nor- 



mal visbn can move pointer 27 about the screen and 
find such items as system ntenu icon 21 or maximize 
icon 25 easfly. However, as can be imagined, blind or 
severely visually impaired people would have a very 
difficult time locating items in a window. Accordingly, 
in the present inventton, sound effects are provided 
to give the user audible feedback about the positton 
of pointer 27. 

In Figure 1, a left/right amplitude scale designat- 
ed generally by the numeral 29 Is depicted along the 
bottom margin of window 1 1 . Scale 29 Is provided only 
for ease of explanation and understanding of the in- 
ventnn and is not actually displayed on the screen. 
In the present Invention, an audible tone is generated 
from a pair of laterally spaced apart transducers. The 
transducers may be either speakers positioned on 
opposite sides of the workstation or headphones 
worn by the user. Scale 29 shows graphically the rel- 
ative left/lrlght amplitudes or balance of the left and 
right channels as a function of the horizontal or 
left/right position of the pointer. Thus, when the poin- 
ter is positioned on the vertical center line of dient 
area 19, the amplitudes of the left and right channels 
are equal to each other and are balanced. As pointer 
27 is moved toward the left, the left channel ampli- 
tude increases while the right channel amplitude de- 
creases. Similarly, as the user moves pointer 27 to- 
ward the right, right channel amplitude increases 
while left channel amplitude deceases. The stereo ef- 
fect provkled by the present Invention enables the 
user almost to "see" the left^right position of the poin- 
ter. 

As the user moves pointer 27 vertically or in the 
top/bottom axis of window 11 , the pitch or frequency 
of the tone varies in stepwise fashion, as depicted by 
the scale 31 displayed along the left hand margin of 
window 11 . Scale 31 shows graphically the stepwise 
arrangement of frequendes as a function of the 
top/bottom positton of the pointer. In the preferred 
embodiment, eight distinct frequencies are provided 
at 300 hertz intervals. The stepwise frequency func- 
tion allows the user to count the steps and thereby 
know how dose pointer 27 is to the top or bottom of 
window dient area 19. The frequency or pitch varia- 
tion enables the user to visualize accurately the 
top/bottom position of pointer 27. Again, scale 31 is il- 
lustrated only for ease of explanation and under- 
standing of the invention, it is not actually displayed 
on the screen. 

With the present invention, the user can tell eash 
ly where pointer 27 is in window dient area 19. By con- 
vention, title bar 15 and action bar 17 are always lo- 
cated at the top of window 11. The choices in action 
bar 17 are always listed left to right starting near the 
upper left hand corner of window 11. Preferably, the 
choices of action bar 17 are announced by text-to- 
speech or recorded speech. Thus, the user can easily 
find the upper left hand corner of dient area 19 and 
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thereby find action bar 17 or system menu icon 21. 
Similarly, minimize icon 23 and maximize Icon 25 are 
always located in the upper right hand corner of win- 
dow 11. which the user can find quickly and easQy. 

Turning now to Rgure 2. there Is shown a block s 
diagram of the system of the present invention. The 
CPU hardware Is contained In dashed rectangle 33. 
Running on CPU hardware 33 Is an operating system 
35 which Includes presentation k)gic 37. A plurality of 
applicattons 39 are shown running on operating sys- io 
tem 35. Video Interface logic and hardware 41 receive 
Information from presentatbn logic 37, which is dis- 
played on a video monitor 43. A mouse 45 and a key- 
board 47 provide user Input to the system. 

The system includes query code 49 which re- is 
celves Infornrmtion firom presentation logic 37 includ- 
ing type of window, position and size of window, and 
current pointer position. Query code 49 provides in- 
formation to sound generation software 51 and hard- 
ware 53. The output from sound generation hardware 20 
53 Is provided to stereo headphones 55 or speakers. 

Referring now to Figure 3. there is shown a block 
diagram of the sound generation software and hard- 
ware of the system of the present invention. Sound 
generation hardware 53 includes a white noise gen- 25 
erator 57 and oscfllator or oscQlators 59. White noise 
generator 57 generates white noise, which sounds 
like a hiss. White noise is actually a mixture of differ- 
ent tones or frequencies in the way that white light Is 
a mbcture of colored light OscQlators 59 add certain 30 
frequency components to the white noise generated 
by white noise generator 57 at a summing circuit 61 . 

The sound generation software outputs include a 
filter center frequency control 63, which operates a 
variable bandpass filter 65. Variable bandpass filter 35 
65 filters out frequency components above and below 
t he filter center firequency and outputs an audio signal 
having a relatively narrow band of frequencies. The 
audio output of variable bandpass filter 65 is per- 
ceived by a listener as either a relatively high pitched 40 
hiss or relatively low pitched hiss depending on the f fl- 
ter center frequency. 

The output from variable bandpass filter 65 is split 
at 67 into left and right channels. A left amplitude corv 
trol 69 controls a variable attenuator 71 in the left 4S 
channel and a right amplitude control 73 controls a va- 
riable attenuator 75 In the right channel. The output 
from variable attenuator 71 is amplified and an output 
amplifier 77 and the audio signal is produced at left 
speaker 79. Similarly, the output from variable attenu- so 
ator 75 is amplified at an output amplifier 81 and pro- 
duced as an audio signal at right speaker 83. 

Referring now to Figure 4, there Is shown a flow- 
chart of a preferred embodiment of the query code of 
the present Invention. First, the pointer position (Xptr. 55 
Yptr) Is queried at block 85. Then, at block 87, the 
Identity and type ^of the window indicated by the poin- 
ter Is queried. Then, the system tests at decision 



block 89 whether the window indicated by the pointer 
Is of the type that uses stereo and balanced sound ef- 
fects. In the present invention, window is defined 
broadly to include not only application windows as de- 
scribed above, but also the background screen, mes- 
sage boxes, dialog boxes, pull-down menus, pop-up 
menus, and the like. In the preferred embodiment of 
the invention, the stereo and balanced sound effects 
are produced only when the pointer Is in the client 
area of an application window. Thus, If the pointer Is 
somewhere other than the client area of an applica- 
t\on window, the sounds are shut off at block 91 if they 
are not used for some other purpose and the system 
returns again to query pointer position at block 85. 

If the pointer is in the dient area of an application 
window, the system queries the windows extents at 
block 93. This amounts to determining the left/right 
limits of the window dient area, which are designated 
Xleft and Xright, respectively, and the top/bottom lim- 
its of the window dient area, which are designated 
Ytop and Ybottom, respectively. Then, at block 95. 
the system calculates the pointer position relative to 
the window extents along the X axis by the formula: 

Px= Xptr-Xleft 
Xright-Xleft 

Then, at block 97, Px, which Is the right channel am- 
plitude, is output to the right amplitude control and 1- 
Px, which is the left channel amplitude, is output to 
the left amplitude control. Next, at block 99, the sys- 
tem calculates the pointer position relative to the win- 
dow extents along the Y axis by the formula: 

Py= Yptr-Ybottom 
Ytop-Ybottom 

Then, at block 101, the system uses Py to calculate 
t he f Dter center firequency by the formula 300 hertz • 
(1+int(Py«8)), which is output to the sound generator. 
The formula of block 101 produces a set of stepwise 
frequendes from 300 hertz to 2.400 hertz, as illustrat- 
ed in Figure 1. After the filter center firequency has 
been output at block 101 , the system returns to block 
85 and again queries pointer positton. 

From the foregoing it may be seen that the sys- 
tem of the present Invention provides a blind or visu- 
ally Impaired user with audio informatton suff Ident to 
enable the user to locate objects in a window. The 
present Invention may also find use among normally 
sighted users who desire additional sensory Input 



Claims 

1 . A met hod of providing a user of a computer sys- 



4 



EP 0 528 743 A1 



8 



tern induding display screen, a pointing device 
for manually positioning a pointer on said screen, 
and a pair of spaced apart speakers, audb Infor- 
mation regarding the position of the pointer in a 
window displayed on the screen, which compris- 5 
esthe steps of: 

monitoring the position of the pointer in 
said window; and, 

generating audio signals from each of said 
speakers, the relative amplitudes of saM audio 10 
signals being proportional to the relative left/right 
position of said pointer in said window. 

2. The method as claimed in dalm 1, wherein the 
frequency of sakl audio signals is proportional to 15 
the relative tDp/lx)ttom position of saM pointer in 
sakl window. 

3. A method of provkJing a user of a computer sys- 
tem induding a display screen and a pointing de- 20 
vice for manually positioning a pointer on said 
screen, audio information regarding the position 

of the pointer in a window displayed on the 
screen, which comprises the steps of: 

monitoring the position of the pointer in 25 
the window; and, 

generating a first audio signal, wherein the 
frequency of said first audio signal is proportional 
to the relative top/bottom position of said pointer 
in said window. 30 



8. A system for providing a user of a computer sys- 
tem, induding a display screen and means for 
manually posittoning a pointer on said screen, au- 
dk) information regarding the positton of said 
pointer in a window displayed on saM screen, 
which comprises: 

a pair of laterally spaced apart speakers; 

means for generating an audio signal from 
each of said speakers; and 

means for varying the amplitude of sakl 
signals generated by said speakers independent- 
ly of each other in response to the relative 
right/left position of said pointer in sakl window. 

9. The system as claimed in daim 8. induding 
means for varying the frequency of sakl signals 
in response to the top/bottom position of said 
pointer in sakl window. 

10. The system as daimed in daim 8, wherein said 
amplitude varying means includes means for 
monitoring the position of said pointer in said win- 
dow. 



4. The method as daimed in daim 3, wherein said 
first audk> emanates from a positkin located to 
one side of said user and induding the step of: 

generating a second audio signal emanat- 
ing from a position located to the opposite skle of 
said user, said second audk> signal having a fre- 
quency substantially equal to the frequency of 
sakl first audio signal. 

5. The method as daimed in daim 4, wherein the 
relative amplitudes of said first and second audio 
signals are proportional to the relative right/left 
position of said pointer in said window. 

8. The method as daimed in daim 5. wherein said 
first audb signal emanates from a right speaker 
and said second audio signal emanates from a 
left speaker, and said amplitude of sakl first audio 
signal increases as said pointer is moved toward 
the right In sard window and said amplitude of 
sakl second audb signal increases as said poin- 
ter is moved toward the left in said window. 
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7. The method as daimed in daim 3, wherein said 
frequency increases in stepwse fashion as said 
pointer is moved toward the top of said screen. 
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